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1. INTRODUCTION. 



This report presents the theoretical base, the methodological approach and the results of an 
investigation which is pan of an ongoing research project concerning prior knowledge of students of 
the Open University of Heerlen (The Netherlands). This research project involves students, opting for 
the course "Economics and Money", and is based on the analysis of the final test-scores for this course 
(ex post facto research). 

An earlier ex post facto-research helped to support the hypothesis that students with an economic s- 
background are better fitted to pass tests for the courses "Economics & Money" and The Balance 
sheet, Profit and Loss account and Administrative procedures" than law-students. Moreover, 
economics-students pass the tests within fewer attempts. To explain these differences, the concept 
"expertise" ("prior knowledge") is presented. "Expertise" is defined as 'the amount and nature of 
available domain-specific or subject oriented knowledge* (Dochy, 1990). 

At the Open University of Heerlen, course developers are well aware of the expertise-problem, e.g. 
that the target student population is very heterogeneous in terras of previous experiences with the 
specific knowledge domains. But the role of expertise in learning knowledge domains and responding 
to tests is rather unknown. Moreover, there is little understanding of the actual expertise or expertise- 
profile of the students involved. Past research clearly identified "expertise" as one of the most 
important factors in facilitating the learning of new knowledge. In this report, an attempt is presented 
to look for "indicators" that help to grasp the expertise of individual students. 

These indicators are expected to correlate to a high extent with the test results of students. "Personal 
and contextual variables" are presented as an operationalisation of these "indicators". More 
information about the use and validity of indicators can be found elsewhere (Segers, et al, 1989, Dochy, 
et al, 1990). This type of indicators is easy to define and information about them is easy to collect It 
is expected they can be considered as good indices of the expertise of individual students. These 
indicators can also help to define specific sub-populations within the experimental group with eg. 
restricted or elaborated expertise. Especially the former sub-population deserves special attention in 
educational settings. Law-students (IS) and economics-students (ES) are considered to have different 
expertise levels. 

Past research has • yet - not been able to detect relevant and valid "indicators". Powell, Conway & 
Ross (1990) revealed e.g. that 'subjective ratings of prior educational level* are better predictors of 
expertise and study success than e.g. 'certificates'. 



2. RESEARCH SETTING ; THE COURSE "ECONOMICS & MONEY* 



2.1. Coarse description 

Along the course "Economics & Money" the following aims are pursued: 

- to acquire an understanding of quantitative and institutional relationships in the economy of 
the Netherlands; 

- to acquire the ability to make a critical analysis of economic arguments; 

- to acquire knowledge of the object of study, the principles and the method of analysis of the 
microeconomy, the macroeconomy, the comparative economy, financial theory, the theory of 
international economic affairs and the theory of economic policy, 

- with the help of the understanding developed during the course, to be able to follow reports 
on major national and international economic developments; 

- to acquire an understanding of the 'subjective' element in practical economics and the 
existence of (sometimes opposing) opinions within the economy; 

- to acquire the ability to manipulate and use economic data as a basis for forming a personal 
opinion on economic phenomena. 

The course is multifunctional, Le. it is part of the compulsory program to gain different higher 
education diplomas (eg. Dutch Law, Public Administration, Informatics and Policy and Management). 
In addition, the course is to be suited for all those who wish to make a first acquaintance with 
economics and science. In addition, the course is a tied choice for students taking diplomas in Business 
Management (higher vocational education and university levels) and Environmental Studies (university 
level). 

The course is a tint level course and represents a nominal study load of 200 hours. The required 
expertise is amy vaguely described: a knowledge of mathematics at higher general secondary education 
level (first degree comparisons; graphs of first degree comparisons; reading and drawing graphs on the 
basis of given figures) and an ability to read English at higher vocational education level. 

22 Course structure 

The course comprises 46 learning units, grouped in eight blocks. The first block is descriptive. Blocks 2 
to 7 represent the theoretical nucleus of the course, and are analytical The final block, block 8, is 
methodological. The revision units in this course are made up of texts, articles and quotations and are 
designed to help the student to check whether he has understood the previous material. These revision 
units contain no supplementary questions or exercises. 

Learning unit 30, belonging to block 5, comprises a computer programme, and offers the student the 
opportunity to gain extensive practice in solving (systems of) comparisons and to gain an 
understanding of how economics models function. 

The tuition plan assumes that four group meetings will be held. In addition, the tuition plan provides 
for weekly discussion sessions. Current practice reveals that there are 8 to 12 group meetings a year 
(De Langen, 1991, personal communication). 



3. RESEARCH DESIGN 
3.1 Hypotheses 

Past research (Dochy, Bouwens, 1990, ex post facto research 1) indicated that the type of student (ES 
or LS) is possibly a relevant indicator of prior knowledge. As a consequence, it seemed that the 
particular course is not fully multifunctional: students aiming at different scientific degrees (student 
type) seem not to obtain comparable results. 

The hypothetical relationship between student type and prior knowledge, with reference to the 
common prior knowledge theories, is further elaborated in the main part of the hypotheses in the 
actual research. 



ERIC 



1. ES obtain higher test scores than LS. 

2. ES obtain better scores for open-ended or multiple-choice questions than LS 

3. ES obtain better test scores than LS for test-items measuring the mastery of procedural knowledge. 

4. ES obtain higher scores than LS for items with a high difficulty index 

5. Personal and contextual variables are relevant indicators of prior knowledge. 

32 Research population 

3JL1 General description of the population 

The course "Economics and Money" has been available since March 1985 and a significant number of 
students were and are enrolled for this course. Until June 1988, 5654 students were enrolled for this 
course. 2541 students finished the course. 3713 of the 1988-students were enrolled for more than two 
courses. 

The global characteristics of this course-population is relatively stable : ± 79% are male and ± 21% 
are female; 41% is younger or equal than 30 years, 40% is between 31 and 40 years, 19% is older, 
about 31% has a higher general or scientific education level (HAVO/VWO), 35% has a higher 
vocational level, 24% have a lower education certificate, 10% have a university degree (Wp). Up to 
25% of the students pass the summative test within six months; another 41% pass the test between the 
seventh and twelfth month. 

3^2 Selection of the sample 

The sample consists of 100 law-students and 100 economics-students. Law-students (LS) and 
economics-students (ES) are defined as students taking at least 2 courses that fit into the compulsory 
program of the diplomas Dutch Law or Economics and subscribed for at least one summative test 
The following background information of the sample helps to describe this population in more detail 
(between brackets, the values are given when considering the entire population) : 24.5% are female 
(21%); 47.2% of the students are 30 years are younger (41%); 52.8% are 31 or older (59%). The 
initial education of the students reveals that 283% of the experimental group have a higher vocational 
education background (35%). Only 7.7% have a university degree (10%). Statistical tests reveal that 
there are no significant differences between this sample and the entire student population. 

33 Research instruments 

Research data were gathered in two ways : a questionnaire helped to describe personal and contextual 
variables in relation to each subject and a test helped to measure the mastery level of subject oriented 
knowledge. 

33.1 Questionnaire 

The administrative information, available in the BASIS-system of the Open University, was 
insufficient to document relevant personal and contextual variables of the research population. In 
order to complete the information from the BASIS-system, a new questionnare was developed 
consisting of multiple choice and open questions. The questions ask for information about the general 
individual background, the actual job, educational level, etc. and were based on the personal and 
contextual variables as described by van Galien-Roodhardt (1987). 
Afrer a first try-out, a final version of the questionnaire was developed. 

When discussing the research results, no seperate analysis of the questionnaire answers will be 
presented. Since the answers help to define personal and contextual variables in the individual 
students, this information will be used to determine the independent variables in our statistical analysis 
of the test results of the individual students. 
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33JZ The subject-oriented knowledge test 
Description and construction of the test 

As mentionned in the introduction, the final test-scores are used as a basis for obtaining information 
about the mastery level of the subject oriented knowledge. A final test consists of two open-ended 
questions (consisting of two or more sub-items) and 30 multiple-choice questions (4 alternatives). 
Besides the subsrvision between open-ended and multiple-choice questions, one can also subdivide the 
questions into two content-dependent subgroups : declarative and procedural questions. 

Declarative questions measure the mastery level of declarative knowledge, Le, appreciation, recognition 
and reproduction of information. Procedural questions measure the ability of the students to apply the 
procedural knowledge, Le production of information and applications (De Corte, et aL,1976). 
In processing the test results for this ex post facto investigation, the dual subdivision between open- 
ended versus multiple-choice questions and declarative versus procedural questions has been taken into 
account It is also to be repeated that not all the students in the experimental group did solve the 
subject oriented knowledge test at the same moment (between November 1985 and October 1988). 
This implies that parallel versions of the test were available. 

Psychometric qualities of the subject-oriented knowledge test 

In ex post facto research designs, the researcher mostly has no impact on the construction of the 
instruments used. As a consequence, it is necessary to check the quality of the test, Le difficulty level, 
reliability and validity. 



Difficulty level 

The difficulty level of test items is not always taken into account when designing tests for Open 
University courses. If taken into account, different procedures have been adopted in the past We 
will dewment the difficulty level of the test items in more detail when discussing the research 
results in relation to hypothesis 4. 

Reliability 

In calculating the mean alfa-coefficient of the 10 parallel test versions, distinction has been made 
between open-ended (OQ) and multiple-choice questions (MQ). 

Table 1 : Mean alfa-coeff Jclents for the subject oriented knowledge test 





alfa 


oa 


.4058 


MQ 


.8053 


Total test score 


.6435 



The reliability-coefficient of the multiple-choice questions is acceptable (alfa > £). The open- 
ended questions are less reliable. This can be explained in different ways. It is possible that the 
open-ended questions are too heterogeneous. Secondly, the scoring of these questions is not done 
in the same and in an objective way. Thirdly, the question-content differs from what the students 
consider to be important. Fourthly, it is possible that in the starting phase of this course, little was 
known about the empirical quality of these newly developed questions. The sample of these 
questions was never researched before and possibly reflects a variety of attempts to design open- 
ended questions. 

Nevertheless, the overall experience always reveals that it is difficult to construct a sample of 
reliable open-ended questions. 

In general, we can state that the overall reliability of the different parallel test versions is acceptable 
in relation to test lenght. 
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Validity 

To define the content validity' of the test, test construction specialists and domain experts were 
interviewed by an independent and unbiased interviewer. During the interview, the construction 
procedure of the test, the relevance of the items and the equilibrity of the sample of testitems were 
discussed. As Carmines & Zeller (1979, p.22) indicate, determining content validity is satisfactory if 
"the universe of the context is accepted as entirely adequate to define the quality to be measured.". 
The results indicate that : the items are considered to be very relevant and the sample of items 
reflects in an equilibrated way the different learning units of the course. Reijnders (1990) reports 
that the relation between tests and learning goals is generally not recognized by students, especially 
suiting students. Our investigation cannot confirm this, although it is clearly possible that this 
phenomenon is a cause of dropout 

Summarizing these findings, the psychometric quality of the subject oriented knowledge test is 

acceptable, with restriction of the reliability of the open-ended questions. The latter fact will be taken 

into account when analysing and interpreting the research results. 

3.4 Research procedure 

The questionnaire was send to the students of the sample group, from which 114 students responded. 
Next, the questionnaire data were related to the final test results for the course "Economic and 
Money". 

The results of 106 subjects were relevant (8 test scores involved of too many missing values). 

The following table presents the frequency distribution of the remaining sample, taking into account 

the moment they passed the summative test 

Tablt 2 : Frequency distribution of LS and ES In relation to the nomtnt they passed the final test 



Date 


Frequency 




ES LS ES+LS 


85.11 


7 13 20 


86.04 


5 5 10 


86.06 


3 4 7 


86.10 


2 5 7 


87.02 


2 6 8 


87.05 


5 2 7 


87.10 


5 5 10 


88.02 


10 2 12 


88.06 


2 9 11 


88.10 


6 8 14 


TOTAL 


47 59 106 



Statistical analysis reveals that there are no significant differences between the initial and final sample 
of students in the experimental group. 

4. DISCUSSION OF THE RESEARCH RESULTS 

4.1 Introduction 

In the analysis of the research findings, the test scores were reviewed in relation to personal and 
contextual variables. One of the more important variables in the actual investigation is the 'diploma 
type' of the students involved (ES or LS). 

42 ES obtain better test scores than LS 

The results of the analysis clearly indicate that the prior knowledge of ES & LS does not differ 
(F« 1.895, p F «.172). This seems in contrast with the research findings of ex post facto 1 (Dochy, 
Bouwens, 1990) and sustains the multifunctional nature of the course "Economics and Money". 
However, multiple classification analysis (MCA) of the above results reveals a very consistent trend : 



Table 3 : MCA table of the total testscores in 
relation to student type (ES or LS) 



ES 


LS 


1.92 


-1.53 



Although not significant, there is clearly a tendency that economy students (ES) obtain a positive mean 
deviation from the mean of the total sample; the contrary is true for law students (LS). 
Before giving a final contusion in relation to the above hypothesis about the interrelation between 
student type and test scores, we further analyse the total test scores by looking at specific sub-scores. 

43 Scores of ES and LS for open-ended or multiple-choice questions 

Analysis of variance of the test scores of LS and ES reveals that there are no significant differences. 
The analysis has been executed for the open-ended (OE), multiple-choice (MC) and total test scores 
(OE + MC). 

Table 4 : Analysis of variance of the testscores and student type (ES or LS) 





OE 


MC 


OE ♦ MC 


F 

Pf 


1.678 
.198 


.870 
.353 


1.895 
.172 



The results of the table suggest that the prior knowledge of ES & LS does not differ. However, 
multiple classification analysis (MCA) of the above results reveals a very consistent trend : 

Tablt 5 : MCA tablt of tht ttttseorts In relation to student typo (ES or LS) 





OE 


MC 


OE ♦ MC 


ES 


1.29 


.63 


1.92 


LS 


-1.03 


-.50 


-1.53 



Economy students seem always to obtain a positive mean deviation from the mean of the total sample; 
law students obtain always a negative mean deviation from the mean. The consistency in this trend is 
striking and suggests that the differences between ES and LS, found in earlier research (Docht, 
Bouwens, 1990, ex post facto 1) are showing themselves. But is has to be repeated that this trend is not 
significant A possible explanation for these less dear research findings can be found in the differences 
between initial expertise level of the student sample involved. Differences can also be due to content 
related aspects, not reflected in the variable 'student type'. 

4.4 Test scores for test-items measuring the mastery of procedural knowledge 

Table 6 gives the results of an analysis of variance of the test results of ES and LS in relation to the 
type of questions. A distinction has been made between declarative and procedural questions. 
We have to repeat that 10 parallel versions of the test have been used; therefore the analysis is 
repeated for each test version. Moreover the analysis is only executed for the multiple-choice 
questions. Table 6 presents the F and p F values and includes data from a further multiple 
classification analysis (MCA) of the results. 
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Table 6 : Analysis of variance qf test results for de&Sarative and procedural 
questions with student type (ES or LS) and MCA * data. 





Declarative quest for 


IS 


Procedural questior 


«s 






P F 




LS 


F 


Pf 


ES 


LS 




1.296 


.270 


• 70 


.42 


/a/ 


/AC 

.495 


- .36 


.20 


fa 


.535 


.485 


1.60 


-1.60 


.074 


.792 


.10 


- .10 


T 


557 
. jfrff 


.3* D 




.43 


.065 


.809 


- .10 


.07 


4 


.016 


.906 


- .43 


.17 


1.190 


.325 


- .71 


.29 


r 
3 


.713 


.431 


3.00 


-1.00 


.196 


.674 


.38 


- .13 


6 


.079 


.790 


- .34 


.86 


1.531 


.271 


- .43 


1.07 


7 


1.235 


.299 


2.60 


-2.60 


6.145 


.038 


1.S0 


-1.30 


8 


.275 


.612 


- .27 


1.33 


.009 


.927 


.or 


- .08 


9 


.580 


.466 


-3.36 


.75 


4.091 


.074 


• .68 


.15 


10 


.210 


.655 


- .57 


.43 


.274 


.610 


.38 


- .29 



There are no significant differences between ES and LS in relation to their scores for declarative or 
procedural questions. The MCA-data do not reveal a consistent trend in the results. As a 
consequence, the hypothesis about the relationship between these two variab'es (student type x type of 
questions) is to be rejected. , 

4.5 Scores for items with a high difficulty index 

There are no significant differences between LS and ES in relation to high or low difficulty indexes of 
the multiple-choice questions (Table 7). Nevertheless, the following trend can be observed : LS get 
higher scores for items with a low p-value. This trend is not observed when comparing items with a 
high difficulty index 



Table 7: Test scores in relation to difficulty index 



COrrMtly 




8 1 8 n 13 IS 17 10 21 23 SW 27 3fi 

multiple choice questions 

-M» lUMtl -*- mm •lnWlU B3«f!lc<llty 1Mb 
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A£ Contextual mud personal variables as relevant indicators of prior knowledge 

The answers on the questionnaire helped to derive 77 other personal and contextual variables (see 
appendix) as potential indicators of prior knowledge. In analyzing the test results, the difference 
between open-ended and multiple-choice questions was taken into account and the students were also 
grouped in accordance to their test scores in a high (33%), medium (33%) and a low group (33%). 
The overall correlation matrix of the personal and contextual variables with the test scores revealed 
little significant correlations, with the exception of some obvious correlations between eg. previous 
educational level and test scores. 

Age 

Age (students <> 30), *s a particular personal variable, does not reveal significant differences in the 
final test scores (OE, MC or OE+MC). 

Previous educational level 

Students with a previous university level (WO) seemed to belong to a very high extent to the high 
group when grouping the students in relation of their final test scores. 

Remarkable is the significant, negative correlation (-.9792") between a secundary educational level 
(VO)with or without mathematics as a main topic. It seems that this diploma level of a majority of 
the students in the experimental group did not comprise mathematics as a main subject This is 
remarkable since "mathematics" is a prior knowledge requirement for the course "Economic and 
. Money". 

Job and job I'vel 

There is a slight significant correlation between test scores for the multiple-choice questions and a 
certain "job level". 81 % of the students with a lower job level (12% of the experimental group) obtain 
high scores for this type of questions. Students working in the sector Trade and Traffic" (9.4 % of the 
sample) obtain mean scores lower than the mean of the total experimental group. 

Students with low test scores : interaction with other indicators 

When focusing the analysis on students with low final test scores (N«39), there are no significant 
correlations with test scores for the open-ended questions (this subgroup of the experimental group 
does not comprise students with a previous university degree). 

Analysis of the scores for the multiple-choice questions does reveal interesting information : 71.4 % of 
the male students have a mean score higher than the mean for the entire group; 72.7 % of the female 
students obtain test scores lower than the mean. 
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5. CONCLUSION 



The results of this investigation can be summarized as follows: 

- The expected differences between ES and LS can not be confirmed, but, nevertheless, there is 
a tendency that economy students perform better than law students. 

- The difficulty level of the test-items does not reveal significant differences between ES and LS 
students. 

- The hypothesis about the potential value of personal and contextual variables as indicators of 
prior knowledge is to be rejected. This is in accordance with the findings of earlier research 
(Cfr. Powell, Conway & Ross, 1990). •Interestingly, the level of previous educational 
experience (formal qualifications), although measured in the study, did not enter the model as 
a significant predictive factor" (Powell et aL, 1990). Although Powell ctal. suggest that 
"subjective ratings" might be indicators of prior knowledge, this research direction is not 
helpful for applicable educational purposes. Subjective perceptions are difficult to influence 
and to change. Moreover it is difficult to relate items to the learning process of subject- 
oriented or domain specific knowledge to be learned or the knowledge already mastered. In 
other words, they are of little help to be used as roars to facilitate or flextbilise the learning 
process. 

- The slight significant correlation between test results and specific personal and contextual 
variables (eg. preliminary educational level) are of little use as indicators of prior knowledge 
since they cannot be manipulated. 

The overall conclusion of this investigation is that research in the field of "prior knowledge" will have 
to be reoriented. In our view, an in-depth analysis and assessment of the virtual domain specific and 
subject oriented expertise of students, is a more promising track. Therefore, the development of special 
domain specific and subject oriented tests which can be related to knowledge profiles will probably be 
of great value. 
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A ppendix 



Variables 



Personal 

student type law/economics; 



age; 



sex. 



Motives 

better functioning in current profession; 

increased opportunities for a new job; 

consider studying as a meaningful way to recreate; 

develop intellectual capacity; 

better social and managerial functioning; 

want to know more about the issues mentioned in the course; 

develop specialism; % 

part of programme; 

not in posession of required certificates for other universities; 

personal situation prevents other study, 

studying whatever,whereever,whenever, 

an appealing course content; 

compose an individual programme of study; 

other than allready mentioned motives. 

Studvtimc 

average studytime per week. 
Education 

with doctoral terminated "WO" education; 
with diploma terminated "HBO" education; 
with diploma terminated "MBO" education; 
with diploma terminated "LBO" education; 
with diploma terminated "VO" education; 
with diploma terminated vocational education; 
with diploma terminated course; 
with certificate terminated Open university course; 
without diploma or certificate terminated training; 
mathematics forms part of the final examinations. 

certification 



diploma taken between three and si* years ago; 
diploma taken between seven and ten years ago; 
diploma teken ber ;n eleven and fourteen years ago; 
diploma taken between fifteen and nineteen years ago; 
diploma taken between twenty and twenty-five years ago; 
diploma taken more than twenty-five years ago. 
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educational institution 

attended no education whatsoever, 

attended education on a daily bases; 

attended part time education; 

attended part time State-aided education; 

attended parttime private education; 

attended education by mail; 

attended a day-release course; 

attended a business course; 

attended a course sponsored by a trade union; 

attended education on a higher educational level; 

attended another not yet mentioned education type. 

nature of profession 

has been active in several fields of activity; 

has been active within the industrial / handicraft field; 

has been active within the trade business / traffic field; 

has been active within the social / medical field; 

has been active within the educational field; 

has been active within the managerial /military field; 

has been active within Government agencies; 

has been active in another, not mentioned above, field. 

active within the trade business / traffic field; 

active within the administrative /financial field; 

active within the social / medical field; 

active within the educational field; 

active within the managerial / military field; 

active within Government agencies 

active within the industrial /handicraft field; 

other 

unemployedjooking for a job or placed on unemployment pay, 

running the household; 

industrial disability; 

because of study without a job. 

position in society 

has been active at a low level position; 
has been active at an average level position; 
has been active at a high level position; 
active at a low level position; 
active at an average position; 
active at a high level position 

payment 

at present a payed job; 
at present a non-payed job. 

workinE hours 

less than 32 hours work a week; 
between 32 and 40 hours work a week; 
40 hours or more work a week. 
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